Skip to content

Adopt python-copier-template 5.5.0 - #216

Open
gilesknap wants to merge 12 commits into
masterfrom
adopt-copier-template
Open

gilesknap wants to merge 12 commits into
masterfrom
adopt-copier-template

Conversation

@gilesknap

@gilesknap gilesknap commented Nov 10, 2026 •

Copy link
Copy Markdown

What and why

Adopts python-copier-template 5.5.0, generated with copier copy so .copier-answers.yml is present and future copier update runs work.

  • The package moves to the template's src/ layout (git mv softioc src/softioc, including the iocStats submodule), so file history is kept and no source file changes.
  • uv.lock replaces Pipfile/Pipfile.lock; metadata moves from setup.cfg to pyproject.toml [project]. setup.py still builds the setuptools_dso extension and sets install_requires (declared dynamic), so the epicscorelibs ABI pin from abi_requires() is kept.
  • epicscorelibs: released wheels and sdists (cibuildwheel, pip) build against the newest epicscorelibs allowed by the [build-system] requires floor and need no edit. A [tool.uv] build-constraint-dependencies pin only affects uv.lock and the dev environment: it keeps the lock stable when new epicscorelibs releases appear. To bump it, edit that one pin, then run uv lock --upgrade-package epicscorelibs. The comment in pyproject.toml and CONTRIBUTING both say this.
  • setuptools_scm replaces _version_git.py.
  • ruff check replaces flake8; pyright, tox and pre-commit (incl. gitleaks) are added.
  • CI uses the template's ci.yml and reusable workflows, with a custom _dist.yml that keeps the cibuildwheel wheel build and test matrix (ubuntu/windows/macos-15-intel, cp311–cp314, now a single cibuildwheel v4.3.5).
  • Docs use pydata-sphinx-theme with myst and still build with -W. They use the template's docs/index.md and four section pages (Tutorials, How-to Guides, Explanations, Reference) with glob toctrees, so the top bar shows the sections and the sidebar's "Section Navigation" lists each section's pages. The existing content pages are unchanged .rst.
  • CONTRIBUTING moves to the template's .github/CONTRIBUTING.md: the template's text plus our sections (clone with submodules, tests, updating dependencies and the epicscorelibs pin, DLS release steps). The docs Reference section includes it.
  • CHANGELOG.rst is frozen at 4.7.2 and kept in the docs as history. Later releases use GitHub Releases with notes generated from PR titles, as in the template; the CHANGELOG release step is removed.
  • Python >= 3.11 (3.7–3.10 dropped).
  • All dependencies are at their latest versions; pip-audit of the exported lock (all groups and extras) and uv audit report no known vulnerabilities.

Supersedes #213: certifi 2026.10.23 is in uv.lock.

Kept minimal on purpose (deviations from the template, to avoid churn)

  • README.rst kept (no README.md); docs/index.md includes its intro with the rst parser.
  • Existing docs .rst content pages kept unrestructured. The section toctrees list pages explicitly where order matters (installation first, why-use-pythonSoftIOC first, API first, and the existing curated how-to order) before the glob, which picks up the rest and any future pages. The contributing page is at docs/reference/contributing.md (the template uses docs/how-to/contribute.md), and the Reference section also keeps the API page, the frozen changelog and the index.
  • No ruff format: no ruff-format pre-commit hook and VS Code format-on-save / organize-imports off.
  • ruff check passes with no code changes: permanent ignores N (public camelCase API) and F401/F403/F405 (star re-exports, as under flake8); temporary ignores I001/UP/SLF001/B/C4/E721 tracked in Fix ruff lint findings deferred from copier template adoption #214.
  • pyright in basic mode with 10 rules off, and no filterwarnings = "error", both tracked in Tighten pyright and pytest warning config deferred from copier template adoption #215.
  • Coverage still off, as decided earlier (Cothread generates segmentation faults when run with Coverage cothread#73); no codecov upload, and the template's coverage paragraph is left out of CONTRIBUTING.
  • Explicit packages = ["softioc"] with package-dir = {"" = "src"} and include-package-data = false, so the wheel's file list is identical to master's (only _version_git.py becomes _version.py).
  • Custom _dist.yml keeping the cibuildwheel matrix and CIBW_* settings; sdist checked with twine check; test-sdist now actually uses each matrix Python.
  • submodules: true on every checkout (iocStats is needed to build).
  • tests/graph.edl excluded from the end-of-file-fixer hook.
  • Apart from the move to src/, the only changes in src/softioc/ and tests/ are the _version import and the template's tests/test_cli.py.
  • Docs publish from master; .github/pages/index.html still redirects to master.

For maintainers to confirm

  • Catalog component_owner: group:default/accelerator-controls and lifecycle production (copied from aioca/epicsdbbuilder).
  • PyPI publishing now runs in a release environment (auto-created with no protection; consider adding reviewers or moving to trusted publishing). It still uses the pypi_token secret, passed explicitly; if the secret is removed the action falls back to trusted publishing.
  • The weekly upstream-master test moves to Wednesday alongside the template's link check (was Monday), and now uses uv. It has not been run yet.
  • The DLS release step now uses uv export --frozen --no-dev --all-extras --no-hashes --no-emit-project > …/softioc-<release>.requirements.txt instead of copying Pipfile.lock; please check this suits the internal process.
  • The epicscorelibs build-constraint pin must be bumped by hand (Renovate won't), with uv lock --upgrade-package epicscorelibs; released wheels are unaffected by it.
  • The dev extra is removed in favour of a dev dependency group (uv sync); cibuildwheel tests use CIBW_TEST_GROUPS: dev.
  • The first release after this needs its breaking changes added to the generated GitHub release notes by hand: the dev extra is replaced by the dev dependency group, and Python 3.7–3.10 support is dropped.
  • Existing dev environments need updating after pulling the src/ move: rm -rf softioc (the old untracked build products and the old iocStats checkout stay behind, and make ruff check . report errors), remove the old top-level epicscorelibs symlink if present, then git submodule update --init and uv sync --reinstall-package softioc (or use a fresh venv) so the editable install points at src/.
  • The sdist now includes the full git tree (tests, docs, iocStats submodule) via setuptools_scm: about 94 KB → 384 KB for the sdist built in CI (non-recursive submodule checkout).
  • The template's license.file (and license classifier) trigger a setuptools deprecation warning: builds will break after 2027-02-18 unless the template fixes it. Not changed here.

How tested

Locally on Linux:

Check Result
uv lock --check OK
uv run ruff check . All checks passed
tox -e type-checking (pyright) 0 errors
tox -e docs (--fail-on-warning) OK
pre-commit run --all-files All pass
Full pytest, Python 3.11 308 passed, 16 skipped (deliberate skips)
pytest subset, Python 3.14 110 passed (test_records, test_cli, test_asyncio, test_deviocstats; 662 warnings, not errors)
uv build (sdist, and wheel built from it) OK
pip-audit of uv export (all groups and extras) / uv audit No known vulnerabilities
  • Wheel file list compared with a wheel built from master: identical apart from _version_git.py → _version.py; _extension*.so, device.dbd, access.acf and the iocStats iocAdmin/Db files are present.
  • Installed wheel and sdist into fresh environments: pythonSoftIOC --version works and an asyncio IOC starts.
  • Docs navigation checked in the built HTML: the top bar has the four sections and each section page's sidebar lists that section's pages.
  • The epicscorelibs bump procedure checked by moving the pin and back: uv lock --upgrade-package epicscorelibs updates the build constraint and softioc's ABI pin in uv.lock, and restores it exactly.
  • copier update --pretend reports "Keeping template version 5.5.0".
  • Plus the CI on this PR.

Found in local review

  • The how-to pages lost their curated order under a pure glob toctree: listed explicitly again.
  • After the src/ move, a stale untracked top-level softioc/ in existing checkouts makes ruff check . fail: the clean-up steps are in the maintainer notes above.
  • PR description said softioc/ instead of src/softioc/, and gave the sdist size without saying it is CI's: fixed.
  • "Headling" typo in CONTRIBUTING: fixed.
  • uv.lock would break on any new epicscorelibs release (dynamic metadata from the build env): build constraint added.
  • The DLS release step copying Pipfile.lock had been dropped: replaced with a uv export.
  • Temporary leniency was not linked to issues: now points at Fix ruff lint findings deferred from copier template adoption #214/Tighten pyright and pytest warning config deferred from copier template adoption #215.
  • rtd-theme-only CSS rules did nothing under pydata: removed (the two-column CSS went with index.rst).
  • VS Code format-on-save would churn whole files: turned off until Fix ruff lint findings deferred from copier template adoption #214.
  • uv export for the DLS release included the dev toolchain: --no-dev added.
  • The build pin was missing from the update docs: added to CONTRIBUTING.
  • secrets: inherit narrowed to an explicit optional pypi_token.
  • The periodic upstream-master job now has read-only permissions.

Found in maintainer review

  • Docs pages were one flat list with an empty "Section Navigation": now the template's section pages.
  • src/ layout, CONTRIBUTING location and CHANGELOG: now as in the template.
  • The epicscorelibs bump instructions were unclear: rewritten.

Found in CI

  • The new uv-based test (ubuntu-latest, 3.14) job segfaulted in cothread (in tests/test_record_values.py). The runner's system python is CPython 3.14.4; the crash reproduces locally with uv-managed 3.14.4, while 3.14.5, 3.14.6 and 3.14.7 pass the full suite. _test.yml now sets UV_PYTHON_PREFERENCE: only-managed so the latest 3.14 patch release is tested. cothread is kept on 3.14 (the cibuildwheel cp314 wheel tests with cothread pass on all three OSes); users on CPython 3.14.4 specifically may hit this crash.

Not in this PR

Do not merge until the maintainers have reviewed; merge commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uybqz8Ac6k38PshyLY6N1C

gilesknap and others added 5 commits November 10, 2026 11:29
Generated with `copier copy` from
https://github.com/DiamondLightSource/python-copier-template (5.5.0) and
reconciled by hand with the existing project:

- Move metadata from setup.cfg into pyproject.toml [project]. setup.py
  still builds the setuptools_dso extension and sets install_requires,
  declared dynamic, so the epicscorelibs ABI pin from abi_requires() is
  kept.
- Replace _version_git.py with setuptools_scm.
- Require Python >= 3.11.
- Replace flake8 with ruff check, using the template's rule selection
  and ignores so the existing code passes unchanged; the temporary
  ignores are to be fixed in a follow-up (#214). ruff format is not used.
- Configure pyright in basic mode with the rules the existing code
  violates turned off.
- Replace code.yml/docs.yml with the template's ci.yml and reusable
  workflows. _dist.yml is custom: it builds the sdist and cibuildwheel
  wheels for cp311-cp314 on Linux, Windows and macOS and tests them as
  before. Checkouts include the iocStats submodule. The weekly test
  against upstream master branches moves to periodic.yml.
- Switch the docs theme to pydata-sphinx-theme with myst, keeping the
  existing .rst pages and index.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz5t8qhYupzSkLkFc5K
Lock all dependencies, including the dev dependency group, at their
latest versions with `uv lock`. pip-audit and `uv audit` report no known
vulnerabilities in the locked packages (certifi is 2026.10.23).

Supersedes #213.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz5t8qhYupzSkLkFc5K
- Add [tool.uv] build-constraint-dependencies pinning epicscorelibs to
  the locked version, so the ABI pin softioc's metadata gets from
  abi_requires() doesn't change (and break `uv run --locked`) whenever a
  new epicscorelibs is released.
- CHANGELOG: note the dev extra is now a dev dependency group.
- CONTRIBUTING: replace the Pipfile.lock copy in the DLS release steps
  with a uv export of the requirements.
- Point the deferred pyright and pytest warning config at #215.
- Drop the sphinx-rtd-theme-only rules from theme_overrides.css, keeping
  the two-column rules that index.rst uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz5t8qhYupzSkLkFc5K
- .vscode: turn off format on save and organize imports, until #214,
  so saving a file doesn't reformat it.
- CONTRIBUTING: export the DLS release requirements without the dev
  group, and document that the epicscorelibs build constraint must be
  bumped by hand.
- Pass the pypi_token secret to _pypi.yml explicitly instead of
  inheriting all secrets.
- Give the periodic master_branch_test job read-only permissions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz5t8qhYupzSkLkFc5K
The ubuntu runner's system python 3.14 is 3.14.4, on which cothread
segfaults in the tests; reproduced locally with uv's 3.14.4, while
3.14.5, 3.14.6 and 3.14.7 pass. Use uv-managed python so the latest
patch release is tested.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz5t8qhYupzSkLkFc5K

@AlexanderWells-diamond AlexanderWells-diamond left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it customary to include the full git tree in an sdist? I don't see why we now include so much more in there

A few miscellaneous points:

  • The Docs haven't been properly integrated into the new format. Every page appears in one flat list. The usual sections of "Tutorial" etc. are not sub-pages, they're just text, so are not working as categories. The "Section Navigation" header on each page is blank.

  • I'd argue that migrating the source files to a src layout should be a part of this PR. It will not create churn as the files themselves won't change, it'll just come up as a git move of the file.

There's probably also a few bits and pieces that we could bring more in line with the template in this PR, but we can start with these ones.

Comment thread pyproject.toml Resolved
Comment thread pyproject.toml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread CHANGELOG.rst Resolved
Comment thread CONTRIBUTING.rst Resolved
gilesknap and others added 6 commits November 11, 2026 07:42
git mv softioc src/softioc, including the iocStats submodule, and point
setup.py, pyproject.toml (package-dir, version_file, pytest, pyright,
ruff) and .gitignore at the new paths.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz8Ac6k38PshyLY6N1C
Released wheels follow the [build-system] requires floor and need no edit;
the uv pin only affects uv.lock and the dev environment. Say how to bump it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz8Ac6k38PshyLY6N1C
Later releases are documented in GitHub Releases with notes generated from
PR titles, as in the python-copier-template. The README links there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz8Ac6k38PshyLY6N1C
The template's text plus our sections (submodule clone, tests, updating
dependencies and the epicscorelibs pin, DLS release steps) in Markdown.
The stale coverage paragraph and the CHANGELOG release step are dropped.
The docs reference page includes it as the template does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz8Ac6k38PshyLY6N1C
docs/index.md and tutorials/how-to/explanations/reference section pages
with glob toctrees replace index.rst, so the pydata theme shows the four
sections in the top bar and each section's pages in the sidebar. Content
pages stay .rst. The two-column CSS only index.rst used is removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz8Ac6k38PshyLY6N1C
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz8Ac6k38PshyLY6N1C
@gilesknap

Copy link
Copy Markdown
Author

Thanks for the review! Taking the points in the review body in turn:

sdist contents: setuptools_scm's file finder includes every git-tracked file in the sdist, and that's the same in every template project. It's what made the sdist grow from about 94 KB to 384 KB (tests, docs and the iocStats submodule). Having the tests in the sdist helps downstream packagers such as conda-forge, and the size is harmless. The wheel contents are unchanged from master apart from _version.py. If you'd prefer a slimmer sdist, I can add a MANIFEST.in that prunes docs/, .github/, .vscode/ and uv.lock.

Docs navigation: fixed in 201df48 (and f71bbcc). The docs now use the template's index.md and four section pages (Tutorials, How-to Guides, Explanations, Reference) with glob toctrees. The top bar shows the four sections and each page's "Section Navigation" lists its section's pages. The existing content pages are unchanged .rst, and they still build with -W.

src layout: done in b9de251, with git mv softioc src/softioc (including the iocStats submodule), so history is kept and no source file changes. setup.py, pyproject and .gitignore follow the new paths. I compared the wheel's file list with one built from master: identical apart from _version.py. Note that existing checkouts need rm -rf softioc, git submodule update --init and uv sync --reinstall-package softioc after pulling.

I'll keep an eye out for other bits that could come into line with the template.

@AlexanderWells-diamond

Copy link
Copy Markdown
Collaborator

There's still one major issue with the docs: the "index.md" does not render properly. I think the conversion has left an odd mix of RST and Markdown syntax in there. Probably due to the inclusion of the README.rst file.

MyST's {include} directive ignores the :parser: option and parsed the
README as Markdown, so the rendered index page showed the RST source
(|ci| badges, the ==== table and .. code:: block) as plain text.

An eval-rst fence with a real docutils include parses it as RST, but
MyST parses that fence in a scratch document, so substitution
definitions in it never resolve and an RST title cannot nest the
Markdown headings that follow. So the README now uses plain image
directives for its badges (which render the same on GitHub and PyPI)
and index.md supplies the H1 itself, including the README from a
marker comment after its title.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uybqz3j1MXkGpuHVYNKjT5
@gilesknap

Copy link
Copy Markdown
Author

Good spot, thanks. The cause was that MyST's {include} directive ignores the :parser: rst option, so README.rst was being parsed as Markdown and the badges, table and code block came out as raw RST.

Fixed in 53d46bb. index.md now includes the README through an eval-rst block with a docutils .. include::, which parses it as RST. Two things don't survive that route, so the README needed small edits: substitution definitions (.. |ci| image::) aren't registered from inside an eval-rst block, so the four badges are now inline .. image:: directives with the same URLs and targets, and the RST title can't nest under MyST's heading tracking, so index.md supplies the # pythonSoftIOC heading and includes the README from a marker comment after its title. README.rst renders the same as before on GitHub and PyPI (checked with docutils and readme_renderer). I built the docs locally and checked the index page: one H1, badges, table and highlighted example, then the four section cards.

The template assumes a README.md, so a follow-up could convert the README to Markdown and let index.md become the template's verbatim file. I've left that out of this PR to keep the diff small.

Docs CI is currently red at the apt-get install graphviz step because GitHub's hosted runners can't resolve the Ubuntu mirrors right now (actions/runner-images#15296). Nothing to do with this change; I'll rerun it when the incident clears.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants